High speed optical image acquisition system with extended dynamic range

ABSTRACT

A high-speed image acquisition system includes a source of light; a sensor for acquiring a plurality of images of the target; a system for determining relative movement between the source and the target; and an image processor for processing the acquired images to generate inspection information relative to the target. The system has an extended dynamic range provided by controlling illumination such that the plurality of images is acquired at two or more different illumination levels. In some embodiments, the high-speed image acquisition system is used to perform three dimensional phase profilometry inspection.

CROSS REFERENCE TO CO-PENDING APPLICATION

[0001] This application is a Continuation-in-Part application of U.S.patent application Ser. No. 09/522,519, filed Mar. 10, 2000, andentitled “Inspection System with Vibration Resistant Video Capture,”which claims priority benefits from U.S. Provisional patent applicationSerial No. 60/175,049, filed Jan. 7, 2000 and entitled “ImprovedInspection Machine.”

COPYRIGHT RESERVATION

[0002] A portion of the disclosure of this patent document containsmaterial which is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but otherwise reserves allcopyright rights whatsoever.

TECHNICAL FIELD

[0003] This invention relates to the acquisition and subsequentconstruction of an extended dynamic range image using a high-speedimaging system.

BACKGROUND OF THE INVENTION

[0004] High-speed optical image acquisition systems are used in avariety of environments to analyze the physical characteristics of oneor more targets. Generally, such systems include an image acquisitionsystem, such as a camera, that can acquire one or more images of thetarget. The images are then analyzed to assess the target. In manycases, the inspection is performed while there is relative motionbetween the target and the image acquisition system. As used herein, a“high-speed” optical image acquisition system is any system thatacquires images of one or more targets while there is relative motionbetween the target and an image acquisition system. One particularexample of such a system includes a phase profilometry inspection systemsuch as that used in some modern solder paste inspection systems.

[0005] Phase profilometry inspection systems are currently used toinspect three-dimensional aspects of target surfaces. The concept ofphase profilometry is relatively simple. A pattern or series of patternsof structured light are projected upon a target at an angle relative tothe direction of an observer. This can be likened to sunlight passingthrough a Venetian blind and falling upon a three-dimensional object,where the object is viewed from an angle that differs from that of thesunlight. The pattern is distorted as a function of the object's shape.Knowledge of the system geometry and analysis of the distorted image orimages can provide a map of the object in three dimensions.

[0006] Generally, phase profilometry systems employ a source ofstructured, patterned light, optics for directing the structured,patterned light onto a three-dimensional object and a sensor for sensingan image of that light as it is scattered, reflected or otherwisemodified by its interaction with the three-dimensional object.

[0007] Phase profilometry inspection can be performed while there isrelative motion between the target and the inspection system. Thisfeature is highly beneficial for automated inspection machines wherethroughput is very important. One example of a phase profilometryinspection system is the Model SE 300 Solder Paste Inspection Systemavailable from CyberOptics Corporation of Golden Valley, Minn. Thissystem uses phase profilometry to measure height profiles of solderpaste deposited upon a circuit board prior to component placement. TheSE 300 System is able to acquire the images that it uses forthree-dimensional phase profilometry while allowing continuous relativemotion between the sensor and the target. This is because a strobeilluminator is operated to provide short exposure times thus essentiallyfreezing motion. Specifics of the illumination and image acquisition areset forth in greater detail in the parent application.

[0008] Solder paste, itself, presents a relatively favorable target inthat it is comprised of a number of tiny solder spheres. Because thereare so many small, spherical reflectors in each solder deposit, in theaggregate, each solder deposit provides a substantially diffuse opticalsurface that can be illuminated and imaged by a sensor configured toreceive diffusely scattered light. The relatively uniform reflectivityof the solder paste deposits facilitates imaging.

[0009] High-speed inspection systems such as the surface phaseprofilometry inspection described above provide highly useful inspectionfunctions without sacrificing system throughput. Theoretically, suchinspection systems would be highly useful for any inspection operationthat requires height information as well as two-dimensional informationfrom a system without adversely affecting system throughput. However,there are real-world hurdles that hinder the ability to extend theadvantages of high-speed inspection beyond targets having relativelydiffuse reflectivity such as solder paste.

[0010] Surfaces such as the underside of a ball grid array (BGA) or chipscale packages (CSP's) are difficult to inspect using current opticalphase profilometry systems. Specifically, the balls on a BGA areconstructed from reheated solder, rendering them substantiallyhemispherical and shiny. Illuminating the hemispherical shiny balls witha structured or directional light source will generate a bright glintfrom a portion of the surface near the specular angle. Conversely, therewill be nearly no light returned by diffuse scattering from theremainder of the hemispherical surface. Imaging the ball with a sensorthat is not configured to deal with this wide dynamic range of returnedlight levels would result in erroneous data.

[0011] When light falls upon any surface, some light will be specularlyreflected and some will be diffusely scattered. Some surfaces (like amirror) are predominantly specular in their reflections, and somesurfaces (like a piece of paper) predominantly scatter light. Imagingany such surface, even highly specular surfaces, may be possible byextending the dynamic range of a sensor configured to receive diffuselyscattered light. One way in which dynamic range has been extended in thepast is by taking or acquiring multiple images of a target withdifferent illumination levels and processing the images to discard imagedata such as saturated pixels or dark pixels. However, all dynamic rangeextension techniques currently known are believed to be applied solelyto systems in which the target and inspection system do not moverelative to each other while the multiple images are acquired. Thus, asdefined herein, such systems are not “high-speed.” In inspection systemswhere throughput cannot be sacrificed by pausing relative movementbetween the inspection system and the target, dynamic range extensionitself is not easy to implement.

SUMMARY OF THE INVENTION

[0012] A high-speed image acquisition system includes a source of light;a sensor for acquiring a plurality of images of the target; a system fordetermining relative movement between the source and the target; and animage processor for processing the acquired images to generateinspection information relative to the target. The system has anextended dynamic range provided by controlling illumination such that aplurality of images is acquired at two or more different illuminationlevels. In some embodiments, the high-speed image acquisition system isused to perform three dimensional phase profilometry inspection.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013]FIG. 1 is a diagrammatic view illustrating operation of ahigh-speed image acquisition system in accordance with an embodiment ofthe present invention.

[0014]FIG. 2 is a diagrammatic view of an inspection system in whichembodiments of the invention are particularly useful.

[0015]FIG. 3 is a flow diagram of a method of performing phaseprofilometry inspection in accordance with an embodiment of the presentinvention.

[0016]FIG. 4 is a flow diagram of another method of performing phaseprofilometry inspection in accordance with another embodiment of thepresent invention.

DETAILED DESCRIPTION

[0017] Embodiments of the invention are applicable to any high-speedoptical image acquisition system. While much of the disclosure isdirected to inspection systems, particularly phase profilometryinspection systems, embodiments of the invention can be practiced in anyimage acquisition system where image acquisition occurs without pausingrelative motion between a target and image acquisition system.

[0018] In the phase profilometry embodiments, the number of images ineach image set is preferably equal to the number of relevant unknowns inthe target. For example, if the target has three substantial unknowns,which might include height, reflectivity and fringe contrast, then threeimages of three differing projected patterns or phases of a singlepattern or a suitable mix would be preferred. Thus, for embodiments withmore than three unknowns, more images would be preferred, and viceversa.

[0019]FIG. 1 is a diagrammatic view illustrating operation of ahigh-speed image acquisition system in accordance with an embodiment ofthe present invention. A representative sensor 100 of a high-speed imageacquisition system is shown with light source 102. Source 102 isflashed, or otherwise energized for a very short duration one or moretimes, during relative movement between sensor 100 and target 104 asindicated by arrow 105. One or more images of target 104 is/are acquiredby sensor 100, corresponding to the flash(es), without pausing therelative motion. After a short interval, inspection system 100 has movedto a new location indicated at 100 a in phantom. At the new location,while moving relative to target 104, illumination source 102 a isflashed again, one or more times at an intensity that is different thanthe first flash or set of flashes. The relative flash intensities of thefirst and second sets of flashes are preferably measured, as will bedescribed later in the specification. Sensor 100 acquires image(s) oftarget 104 corresponding to the subsequent set of flashes. Sensor 100measures, or otherwise determines, the relative sensor movement betweenthe first and second flashes, or sets of flashes and uses thatinformation to register or cosite the two acquired image sets. Inembodiments where the number of images in each set is one, the detectorarray used in sensor 100 is preferably an interline transfer CCD arrayor CMOS array.

[0020] Once the image sets are registered, a composite image withextended dynamic range can be created. Extended dynamic range isprovided because if one image set has saturated or black portions, datafrom the other image can be used and normalized based on the measuredillumination intensities or the measured ratio between the twointensities.

[0021] Target 104 may be a two-dimensional or three-dimensional object.Further, the illumination generated during either or both flashes can beof any suitable wavelength, including that of the visible spectrum.

[0022]FIG. 2 is a diagrammatic view of an exemplary inspection system200 in which embodiments of the present invention are useful. System 200includes image processor 202 and sensor 201. Sensor 201 includesprojector 203, and imager 206. Projector 203 includes illuminator 210,grating 212, and optics 214. Illuminator 210 is preferably a gasdischarge flashlamp, such as a high intensity xenon arc lamp, but can beany light source capable of providing short duration light pulses, suchas a pulsed laser or LED. Grating 212 imparts a pattern on the lightpassing therethrough, and optics 214 focuses the patterned light upontarget 216 on circuit board 218. Imager 206 includes optics 219, whichfocus the image of target 216 upon array 220, which is preferably aframe transfer CCD array because each set of images includes more thanone image in this embodiment. As the target moves in the Y directionrelative to sensor 201, the relative position of the target is observedby processor 202 via position encoders (not shown). Processor 202 iscoupled to projector 203 such that processor 202 triggers projector 203to project a patterned image upon target 216 on circuit board 218according to the observed position. Target 216 may be, for example, asolder paste deposit, an electrical component or a package of anelectrical component. As motion continues in the Y direction, two moreimages are acquired to complete the set of three images used in thisembodiment. The target motion relative to sensor 201 provides the phaseshift required to generate the p, p+120 and p+240 degree images used inthis embodiment to form a three-dimensional image of the target.

[0023] CCD array 220 provides data representative of the acquired imagesto image processor 202 which computes a height map of target 216 basedupon the acquired images. The height map is useful for a number ofinspection criteria.

[0024]FIG. 3 is a flow diagram of an embodiment of the invention whereextended dynamic range phase profilometry is performed using ahigh-speed inspection system. Method 300 begins at block 302, where afirst set of at least one image is acquired while the acquisition systemis moved relative to the target, and while the light source is flashedat a first illumination level. The first set of preferably three imagesis acquired in an extremely small amount of time in order to increasevibration immunity. For example, the first triplet can be obtained in 1millisecond or less, as disclosed in the parent application. Each imageof the first set is acquired while a different phase or pattern isprojected onto the target. Preferably, a sinusoidal fringe pattern isused, and each image is separated from the other by a lateral distancethat corresponds to approximately 120 degrees of phase change. Forexample, the images can be acquired at p, p+120, and p+240 degrees,where p is some arbitrary phase.

[0025] Typically, throughout inspection, there is continuous relativemotion between sensor 201 and target 216. At block 306, a second set ofat least one image is obtained while target 216 is illuminated at asecond illumination level that differs from the first illuminationlevel. Preferably, the second set consists of three images obtained withdiffering phases of a sinusoidal fringe pattern. The second set is alsoobtained within an interval of preferably one millisecond, but need notbe within any specific period from the acquisition of the first set, aslong as the difference is not so great that the two sets cannot berelated to one another. Preferably, however, if the first set of imagesis acquired at p, p+120, and p+240 degrees, the second set is acquiredat p+360n, p+360n+120 and p+360n+240 degrees, where n is an integer. Inthis manner, the images of the fringe pattern on target 216 will beidentical except for differences due to the intentional illuminationlevel change and the relative motion between sensor 201 and target 216corresponding to 360n degrees.

[0026] At block 308, the first and second sets of at least one image areregistered. This can be performed by adjusting one or both sets ofimages based upon measured, or otherwise known movement that occurredbetween acquisition of the first and second sets. Additionally, in someembodiments, it is possible to apply image processing techniques to thesets of images themselves to recognize patterns or references in theimages and manipulate the images such that the references are aligned.Further still, measured motion between the first and second sets ofacquisitions can be used to perform a first stage manipulation where oneor both sets are manipulated for a coarse alignment. Then, the resultsof the coarse alignment are processed to identify patterns or referencesin the coarse-aligned images in order to finely adjust the images. Oncethe images are registered, the dynamic range can be extended asdiscussed above.

[0027] Generally, the brighter set of images (those obtained using thegreater of the illumination levels) should be usable until saturation,presuming only that the detector and analog circuitry are linear, or atleast stable (in which case non-linearity can be calibrated out).However, any suitable technique or algorithm can be used.

[0028] Some embodiments of the present invention require that thedifference in illumination levels be precisely controlled. Although itmay be technically possible to provide an illumination circuit with aprecisely-controlled intensity, it is preferred that the illuminationlevels be only approximately controlled, but then measured to therequired precision during use. As an example, the brighter illuminationlevel may be one hundred times brighter than the dimmer level. In oneembodiment, illumination levels are measured during use and used tocorrect for variations from one set to the next with sufficientprecision. Preferably, the required precision is approximately equal tothe smallest relative error detectable by the analog to digitalconverter. Thus, with a 10-bit converter, it would be sufficient tomeasure the relative light levels to a precision of one part in 210;with a ratio of light levels of 100 to 1, the resulting dynamic rangewould be 100 times 2¹⁰ or about 100,000 to 1.

[0029] This approach is preferred due to the difficulty of controllingthe strobe discharge with the required precision.

[0030] Alternatively, each set of images can be constructed into heightmaps before dynamic range extension and then three dimensional pixels(zetels) from the bright map can be replaced with zetels from the dimmap depending on whether there was saturation in the images used toconstruct the bright map. The strobe intensities must still be wellcontrolled or measured, since such intensity errors affect the heightreconstruction, but the task is easier since only the nominally-equalstrobe intensities within the set used to construct each height map needbe measured or controlled; the magnitude of the large ratio between setsneed not be known or controlled accurately.

[0031] Other normalization techniques are also possible. For instance,each image in the set can be normalized (that is, divided) by itscorresponding measured strobe intensity. Then the normalized pixels canbe freely chosen from one set or the other, based on their positions intheir respective dynamic ranges, with the height reconstruction donelater on the composite image set thus produced. This technique relies onthe fact that phase reconstruction is not affected by a scaling that isapplied uniformly to all the phase images. The suitability of aparticular technique may depend on the details of the hardware orsoftware used to make the selection.

[0032] At block 310, a height map is reconstructed using the compositeset of image data from step 308. The manner in which the height map isgenerated can include any of the well-known phase profilometry heightreconstruction techniques. Once the height map is reconstructed, step312 is executed where profilometry inspection is performed on thereconstructed height map. This inspection can include extracting summarymeasures such as height, volume, area, registration, coplanarity,maximum height range, surface contours, surface roughness . . . etc.

[0033]FIG. 4 is a flow diagram of another method of performing phaseprofilometry inspection in accordance with another embodiment of thepresent invention. Method 400 begins similarly to method 300, with steps402, 404, and 406 providing for the acquisition of first and second setsof images while the image acquisition system moves relative to a targetsurface. However, method 400 differs from method 300 in one importantrespect. Specifically, after step 406, method 400 provides step 408where height maps, one for each set of acquired images, arereconstructed. Then, at step 410, the height maps are reviewed todetermine if they include portions that were based on bad pixel data,such as saturated pixels. If a first height map includes such a portion,a corresponding portion of the second height map is reviewed todetermine if its pixel data for that portion is usable. If so, theportion of the first height map is replaced with the correspondingportion of the second height map. Since the maps are generated prior tostep 410, the replacement operation need not take into account thedifference in illumination intensity between the images used for thefirst height map and that of the second height map as long as thevariation of illumination intensity within a given set is the same, orcan be measured and corrected. At step 412, the final height map is usedin profilometry inspection. Although FIG. 4 illustrates acquiring thesecond set before performing any height construction, it is alsopossible to reconstruct the first height map before acquiring the secondset of images.

[0034] Although the present invention has been described with referenceto preferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

What is claimed is:
 1. A high-speed image acquisition system comprising:a sensor movable relative to a target; an illuminator adapted to directillumination upon the target at a plurality of illumination levels;wherein the sensor obtains a first set of at least one image at a firstillumination level during relative motion between the sensor and thetarget, and the sensor obtains a second set of at least one image at asecond illumination level of the plurality of illumination levels duringrelative motion between the sensor and the target; wherein the first andsecond illumination levels differ from each other by a known ratio, andwherein an unusable portion from one set of at least one image isreplaced based upon the other set of at least one image and the knownratio; and an image processor coupled to the sensor and adapted tocreate an extended dynamic range image based at least in part uponcositing the first set of at least one image with the second set of atleast one image.
 2. The system of claim 1, wherein the illuminator is astroboscopic illuminator.
 3. The system of claim 2, wherein theilluminator is a gas discharge strobe.
 4. The system of claim 1, whereinthe illuminator is movable relative to the target.
 5. The system ofclaim 1, and further comprising a position measurement system coupled tothe sensor, the position measurement system providing an indication tothe image processor relative to displacement occurring betweenacquisition of the first and second sets.
 6. The system of claim 5,wherein the image processor registers the first and second sets basedupon the indication.
 7. The system of claim 6, wherein the imageprocessor registers the first and second sets based upon imageprocessing of the first and second sets.
 8. The system of claim 1,wherein the target is an electrical component.
 9. The system of claim 1,wherein the target is a package of an electrical component.
 10. Ahigh-speed inspection system comprising: a source disposed to projectlight upon a target at a plurality of illumination levels; an imagingsystem for acquiring a plurality of sets of at least one image, whereina first set is related to a first illumination level of the plurality ofillumination levels, and a second set is related to a secondillumination level of the plurality of illumination levels, duringrelative movement between the imaging system and the target; and aprocessor for processing the acquired images to generate inspectioninformation with an extended dynamic range.
 11. The system of claim 10,wherein the source is adapted to project patterned illumination.
 12. Thesystem of claim 11, wherein the pattern is a sinusoidal fringe pattern.13. The system of claim 12, wherein the first set of at least one imageincludes three images acquired at phases that are substantially p, p+120and p+240 degrees of the sinusoidal function.
 14. The system of claim13, wherein the first set is acquired within one millisecond.
 15. Thesystem of claim 13, wherein the second set of at least one imageincludes three images acquired at phases that are substantially p+360n,p+360n+120, and p+360n+240 degrees of the sinusoidal function, where nis an integer.
 16. The system of claim 15, wherein the second set isacquired within one millisecond.
 17. The system of claim 10, wherein thesource is a strobe.
 18. The system of claim 17, wherein the sourceincludes a gas discharge flashlamp.
 19. The system of claim 18, whereinthe source is a xenon arc flashlamp.
 20. The system of claim 10, whereinthe first and second sets of images are registered using imageprocessing techniques.
 21. The system of claim 10, and furthercomprising a system for measuring displacement between the imagingsystem and the target during the interval between acquisition of thefirst and second sets, and wherein the first and second sets areregistered based at least in part upon the measured displacement. 22.The system of claim 10, and further comprising an illumination sensorfor sensing the first and second illumination levels to provide a ratiotherebetween.
 23. The system of claim 22, wherein the processorregisters the first and second sets to create a composite set.
 24. Thesystem of claim 22, wherein portions of images in one set are replacedby portions of images in the other set by normalizing the portions inthe other set using the ratio.
 25. The system of claim 10, and furthercomprising an illumination sensor for sensing the first and secondillumination levels to provide a difference therebetween.
 26. The systemof claim 25, wherein the processor registers the first and second setsto create a composite set.
 27. The system of claim 25, wherein portionsof images in one set are replaced by portions of images in the other setby normalizing the portions in the other set using the ratio.
 28. Amethod of inspecting a target with a high-speed inspection system, themethod comprising: obtaining a first image of the target at a firstillumination level with a sensor while the sensor and target moverelative to each other; obtaining a second image of the target at asecond illumination level with the sensor while the sensor and targetmove relative to each other, wherein the first and second illuminationlevels differ from each other by a known amount; replacing an unusableportion of one of the first and second images with a correspondingportion of the other of the first and second images by normalizing thecorresponding portion based upon the known amount of difference inillumination levels, to create a composite image; and processing thecomposite image to generate inspection information relative to thetarget.